Recognize log level in Elasticsearch JVM logs #34159

danielmitterdorfer · 2023-01-03T08:12:32Z

What does this PR do?

Elasticsearch will add a log level to its JVM logs to allow users to help them detect errors / warnings more easily in elastic/elasticsearch#92382. With this commit we detect this new field if present and continue to recognize the prior log format which does not contain a log level.

Why is it important?

This change is required to keep compatibility with Elasticsearch's JVM log output.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates elastic/elasticsearch#92382
Closes #34054

Elasticsearch will add a log level to its JVM logs to allow users to help them detect errors / warnings more easily. With this commit we detect this new field if present and continue to recognize the prior log format without a log level. Relates elastic/elasticsearch#92382 Closes elastic#34054

botelastic · 2023-01-03T08:12:40Z

This pull request doesn't have a Team:<team> label.

mergify · 2023-01-03T08:13:08Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @danielmitterdorfer? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

danielmitterdorfer · 2023-01-03T08:18:10Z

There is one thing where I'm specifically looking for guidance from reviewers. In line with elasticsearch.gc.tags, I've put the log level in elasticsearch.gc.level but I wonder whether it makes more sense to put it into log.level instead.

elasticmachine · 2023-01-03T08:20:23Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-09T06:58:53.995+0000
Duration: 72 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7203
Skipped	738
Total	7941

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

danielmitterdorfer · 2023-01-03T14:39:21Z

x-pack/filebeat-pythonIntegTest is currently failing but none of the failing tests are related to the Elasticsearch module (the failures are in cyberarkpa and cisco).

ruflin · 2023-01-04T08:00:35Z

Lets make sure, any change we add here also makes it into the integrations package: https://github.com/elastic/integrations/tree/main/packages/elasticsearch/data_stream/gc

danielmitterdorfer · 2023-01-04T09:55:52Z

Lee has already opened an issue for that: elastic/integrations#4842. :)

leehinman

I think log.level is the way to go, otherwise everything looks great.

danielmitterdorfer · 2023-01-04T15:12:07Z

I think log.level is the way to go, otherwise everything looks great.

Great, thanks for having a look @leehinman . Is this as simple as just changing the grok pattern from

(\[%{DATA:elasticsearch.gc.level}%{SPACE}\])?

to

(\[%{DATA:log.level}%{SPACE}\])?

or am I missing something?

danielmitterdorfer · 2023-01-04T15:42:35Z

I've pushed the rename now in 24cf0bb. @leehinman can you please take another look?

miltonhultgren · 2023-01-04T15:46:35Z

Lee has already opened an issue for that: elastic/integrations#4842. :)

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

danielmitterdorfer · 2023-01-04T15:52:09Z

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

@miltonhultgren I would hope it's not much effort once this change is in (basically copy & paste what's done here)? I'd like to see this change being released with 8.7.0. I can attempt to do this myself but will require some guidance.

leehinman

LGTM

leehinman · 2023-01-04T18:19:54Z

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

@miltonhultgren I would hope it's not much effort once this change is in (basically copy & paste what's done here)? I'd like to see this change being released with 8.7.0. I can attempt to do this myself but will require some guidance.

Let me know if you want help.

miltonhultgren · 2023-01-05T12:22:19Z

@danielmitterdorfer @leehinman Okay, I had to confirm with our expert about what would be affected but it seems like a small change so I think we can handle that, I'm putting it into our queue!

danielmitterdorfer · 2023-01-09T08:15:53Z

@miltonhultgren thanks for taking care of the changes on the integrations side!

@leehinman The changes are now approved and all PR checks are green but I am not authorized to merge the PR. Could you please merge the PR on my behalf? Thank you!

* Recognize log level in Elasticsearch JVM logs Elasticsearch will add a log level to its JVM logs to allow users to help them detect errors / warnings more easily. With this commit we detect this new field if present and continue to recognize the prior log format without a log level. Relates elastic/elasticsearch#92382 Closes #34054

danielmitterdorfer added enhancement Filebeat Filebeat v8.7.0 Module:elasticsearch Elasticsearch Beats modules labels Jan 3, 2023

danielmitterdorfer requested a review from leehinman January 3, 2023 08:12

danielmitterdorfer requested a review from a team as a code owner January 3, 2023 08:12

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 3, 2023

mergify bot assigned danielmitterdorfer Jan 3, 2023

Add changelog

6eac538

Run make update

bf677ea

danielmitterdorfer requested a review from a team as a code owner January 3, 2023 08:42

danielmitterdorfer requested review from belimawr and removed request for a team January 3, 2023 08:43

Update expected offsets

a290b32

Merge branch 'main' into es-jvm-log-level

b89d68a

danielmitterdorfer mentioned this pull request Jan 4, 2023

Add log level for JVM logs elastic/elasticsearch#92382

Merged

leehinman requested changes Jan 4, 2023

View reviewed changes

elasticsearch.gc.level -> log.level

24cf0bb

danielmitterdorfer requested a review from leehinman January 4, 2023 15:42

danielmitterdorfer and others added 2 commits January 4, 2023 16:47

Remove obsolete field from docs

90893f3

Merge branch 'main' into es-jvm-log-level

ed78bd3

leehinman approved these changes Jan 4, 2023

View reviewed changes

Merge branch 'main' into es-jvm-log-level

1a29ef0

miltonhultgren approved these changes Jan 9, 2023

View reviewed changes

leehinman merged commit 27b8103 into elastic:main Jan 9, 2023

danielmitterdorfer deleted the es-jvm-log-level branch January 9, 2023 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recognize log level in Elasticsearch JVM logs #34159

Recognize log level in Elasticsearch JVM logs #34159

danielmitterdorfer commented Jan 3, 2023 •

edited

Loading

botelastic bot commented Jan 3, 2023

mergify bot commented Jan 3, 2023

danielmitterdorfer commented Jan 3, 2023

elasticmachine commented Jan 3, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

danielmitterdorfer commented Jan 3, 2023

ruflin commented Jan 4, 2023

danielmitterdorfer commented Jan 4, 2023

leehinman left a comment

danielmitterdorfer commented Jan 4, 2023

danielmitterdorfer commented Jan 4, 2023

miltonhultgren commented Jan 4, 2023 •

edited

Loading

danielmitterdorfer commented Jan 4, 2023

leehinman left a comment

leehinman commented Jan 4, 2023

miltonhultgren commented Jan 5, 2023

danielmitterdorfer commented Jan 9, 2023

Recognize log level in Elasticsearch JVM logs #34159

Recognize log level in Elasticsearch JVM logs #34159

Conversation

danielmitterdorfer commented Jan 3, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

botelastic bot commented Jan 3, 2023

mergify bot commented Jan 3, 2023

danielmitterdorfer commented Jan 3, 2023

elasticmachine commented Jan 3, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

danielmitterdorfer commented Jan 3, 2023

ruflin commented Jan 4, 2023

danielmitterdorfer commented Jan 4, 2023

leehinman left a comment

Choose a reason for hiding this comment

danielmitterdorfer commented Jan 4, 2023

danielmitterdorfer commented Jan 4, 2023

miltonhultgren commented Jan 4, 2023 • edited Loading

danielmitterdorfer commented Jan 4, 2023

leehinman left a comment

Choose a reason for hiding this comment

leehinman commented Jan 4, 2023

miltonhultgren commented Jan 5, 2023

danielmitterdorfer commented Jan 9, 2023

danielmitterdorfer commented Jan 3, 2023 •

edited

Loading

elasticmachine commented Jan 3, 2023 •

edited by jenkins-beats-ci bot

Loading

miltonhultgren commented Jan 4, 2023 •

edited

Loading